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The effects of three information distribution strategies on student performance and satisfaction were 
investigated. Data analysis foimd that distributing materials in class resulted in higher student 
performance on an end-of-the-semester knowledge test than when materials were distributed via a course 
management system at the beginning of the semester or one week before each class. No significant 
differences were found on total course points or student reactions. Implications for theory, practice, and 
research are discussed. 
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In recent years colleges and universities have made substantial investments in tools to facilitate the integration of 
technology into instruction. One of the tools being widely embraced by institutions of higher learning is web-based 
course management systems (CMSs). A 2003 report indicated that more than 80 percent of U.S. colleges and 
universities were utilizing CMSs (Harrington, Gordon, & Schibik, 2004). The cost of CMSs is significant, with 
many institutions spending millions of dollars on CMS development, licensing, faculty training, and student support 
services (Sausner, 2005). 

Surely, this substantial level of expenditure is expected to reap rewards in terms of improving the quality of 
student learning and performance. Indeed, Sausner (2005) quotes the Vice President and CIO of the University of 
Cincinnati, Fred Siff, as stating that “the money [spent on CMSs] is inconsequential compared to the value and the 
importance because you’re finally affecting the classroom.” This statement sounds impressive, but makes one 
wonder how these web-based course management systems are affecting the classroom and whether those effects 
have a positive impact on student performance and satisfaction. 

Course management systems typically have the capacity to assist in a wide range of instructional tasks, 
including delivering course content, facilitating interaction among instructors and students, and evaluating learner 
performance (Bonk, Cummings, Hara, Fischler, & Lee, 1999). While CMSs have such capabilities, it has been 
widely reported that the most frequently used component of a CMS is its content presentation tools (Morgan, 2003; 
Nicoll & Laudato, 1999). Instructors typically use content presentation tools to distribute syllabi, class notes, 
participant handouts, and grades to students via the Web. 

However, the practice of distributing such course information on-line may conflict with several principles of 
effective instruction. One principle is that for optimal levels of learning to occur, regardless of the type of learning 
outcome desired, new information should be arranged and presented in such a way so that instruction starts with 
familiar material and builds new knowledge through a step by step progression of facts and connections to arrive at 
new understandings (Gagne & Medsker, 1996; Knowles, 1986). This principle appears to be violated when class 
materials are distributed on-line because decisions regarding the amount and complexity of information that is 
presented must be made before a lesson commences. 

A second, related principle of effective instruction that appears to conflict with the distribution of course 
materials via a CMS is that the timely presentation of new information during instruction helps learners focus on the 
learning tasks at hand and promotes their motivation to engage in planned learning activities (Joyce & Weil, 1996). 
Yet, on-line course materials are made available to learners anywhere from a day to a semester before the class 
sessions in which the materials are presented. It is possible that such advanced access to instructional content may 
result in student confusion and/or boredom and, as a consequence, may be counterproductive to student performance 
and satisfaction. 

Most empirical studies of CMSs have used survey research designs and provide little assistance in resolving 
these possible conflicts between instructional theory and practice. Specifically, most of the survey studies have 
found that while students like the convenience of having course materials on-line (Bonk et al., 1999; Morgan, 2003), 
they do not believe that on-line access to course materials impacts their learning or performance (Young, 2004). 
Few studies, if any, have used experimental methods to examine the impact of distributing course materials via 
CMSs on academic performance and satisfaction. 
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Further investigation of this topic is of great significance to undergraduate and graduate HRD programs in colleges 
and universities. Decisions regarding the delivery of instruction must be based not only on the technological 
capabilities of CMSs, but also on the impact that they have on student performance and satisfaction. Empirical 
studies that provide such information will provide greater insights that can be used to evaluate and possibly 
reconsider the ways that HRD instructors use CMS tools in their courses. 

Theoretical Framework: Disconnects between Instructional Theory and CMS Practices 

For over 40 years, Gagne’s nine events of instruction have served as a framework for designing effective 
instructional programs (Kemp, Morrison, & Ross, 1998; Smith & Ragan, 1999). The nine external events of 
instruction and the internal learning processes they promote (noted in parentheses) are: (1) Gaining attention 

(reception), (2) Informing learners of the objective (expectancy), (3) Stimulating recall of prior learning (retrieval to 
working memory), (4) Presenting the content (selective perception), (5) Providing learning guidance (semantic 
encoding), (6) Eliciting performance (responding), (7) Providing feedback (reinforcement), (8) Assessing 
performance (retrieval and reinforcement), and (9) Enhancing retention and transfer (retrieval and generalization). 
While instructional methods, such as expository and inquiry methods, sequence these instructional events in 
different orders, it is widely accepted that effective instruction does involve the use of all nine events. 

An important instructional principle underlying the effective design of these instructional events concerns the 
sequencing and organization of new information (Gagne & Medsker, 1996; Smith & Ragan, 1999). Specifically, 
this instructional principle dictates the arrangement and presentation of new information in such a way so that 
instruction starts with familiar material and builds new knowledge through a step by step progression of facts and 
connections to ultimately arrive at new understandings. This instructional principle infers that the amount and 
complexity of instructional content should be continuously adjusted as a lesson progresses in response to the pace 
and degree of learning that is occurring in the classroom. This type of information presentation strategy is likely to 
promote high levels of motivation and achievement of learning outcomes. Likewise, too complex or too much 
information presented too soon may inhibit a learner’s ability and/or desire to connect new information to current 
knowledge and, ultimately, diminish his/her ability to make meaning out of the learning experience. 

A second principle of effective instruction is that the timely presentation of new information is critical to the 
achievement of a program’s learning objective (Smith & Ragan, 1999). This principle is anchored in the notion that 
the instructor or another instructional medium, such as a computer simulation, should decide when new information 
is presented and this decision should be made in concert with the instructional plan that has been designed to lead 
learners through the nine events of instruction (Knowles, 1986). As a consequence, many instructional theorists 
assert that when the instructor or instructional medium controls the time at which new information and materials are 
presented during a lesson, it enhances the focus of learners on the task at hand and promotes their motivation to 
engage in the planned learning activities (Joyce & Weil, 1996). Support for this instructional principle can be found 
in previous instructional design studies that found that providing highly structured and clearly ordered course 
materials in advance tends to foster superficial, memory-level encoding, but impedes the far transfer of material to 
new learning tasks (Wilson & Cole, 1992). A similar issue may arise when learners access course materials via 
CMSs. It is possible that such advanced access to course materials may provide so much organization and structure 
to new information that it actually impedes active processing of that new information and, thereby, leads to more 
superficial levels of student learning and lower levels of motivation. 

These two conflicts between instructional theory and practice drive the need to look more closely at the 
capabilities of CMSs and how they are commonly used to promote student learning, performance, and satisfaction in 
post-secondary education. Bonk et al. (1999) created a ten- level continuum for integrating the Web into instruction 
in higher education in an attempt to clarify the pedagogical choices that instructors should consider when using 
CMSs. Levels 1 through 5 of the continuum focus on tools for information-sharing: Level 1-Marketing the course 
and the course syllabus, Level 2-Student exploration of Web resources, Level 3-Student-generated resources posted 
on the Web, Level 4-Course resources on the web, and Level 5-Repurpose Web resources. All of these first five 
levels provide students with alternative sources for course information rather than provide features of the course that 
cannot be acquired elsewhere. Levels 6-10 of the continuum are distinguished from the first five levels in that the 
learning activities and tasks embedded in these higher levels contain requirements of the course that are not provided 
elsewhere. These levels include: Level 6-Substantive and graded Web activities, Level 7-Course activities 
extending beyond class, Level 8-Web as an alternative delivery system for resident students, Level 9-Entire course 
on the Web for students located anywhere, and Level 10-Course fits within larger programmatic Web initiative. 

Most CMSs possess the capabilities to provide instructional experiences at all ten levels of Bonk’s continuum. 
However, the most commonly used component of a CMS is its content-presentation features, that is, Levels 1 
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though 5 of the web-integration continuum. For example, recent surveys at the University of Wisconsin (Morgan, 

2003) and the University of Pittsburgh (Nicoll & Laudato, 1999) found that faculty used the content-presentation 
features (e.g. lecture notes, quizzes, and course announcements) much more frequently than the interactive features 
of CMSs (e.g. on-line discussions and chats). And, a recent nationwide survey of 4,373 students at 13 U.S. colleges 
echo the findings from these two university surveys (Young, 2004). 

In large part, the rapid and widespread adoption of CMS content-presentation tools in higher education results 
from two prevailing beliefs, one held by students and the other by educational administrators. From the students’ 
perspective, a key benefit of on-line access to course materials is convenience (Nicoll & Laudato, 1999; Young, 

2004) . Students report that on-line access allows them to download course materials if and when they choose to do 
so and this level of control enhances their ability to prepare for course sessions and organize course materials. 

Educational administrators have also embraced the use of content-presentation tools in CMSs. In large part, 
institutional support for this feature of CMSs is predicated on the notion that on-line access to course materials 
promotes student learning and satisfaction (Nicoll & Laudato, 1999). A chief assumption underlying this notion 
appears to be that on-line access to course materials helps students prepare for and engage in class lectures and 
activities (Morgan, 2003). 

Interestingly, while survey research studies have found that faculty and students both believe that students like 
the convenience of having course materials on-line (Bonk et ah, 1999; Morgan, 2003), both groups also report that 
they do not believe that on-line access to course materials positively impacts student learning or performance 
(Flarrington et ah, 2004; Young, 2004). Few studies, if any, have used experimental research designs to study the 
effects of CMSs on academic performance and satisfaction. 

As a consequence, the question of whether the distribution of course materials on-line enhances academic 
performance is presently unknown. Therefore, the purpose of this study was to examine the effects of information 
distribution strategies in a CMS on student performance and satisfaction in a university course. Three information 
distribution strategies were investigated: (a) on-line access to participant materials via a CMS at the beginning of the 
semester in which the materials were presented, (b) on-line access to participant materials via a CMS one week prior 
to each class session in which the materials were presented, and (c) no on-line access to participant materials — all 
materials distributed in class. Two research hypotheses were investigated: (1) In-class distribution of participant 
materials will result in higher levels of student performance than either type of CMS information distribution 
strategy, and (2) In-class distribution of participant materials will result in greater student satisfaction with the 
instructional experience than either type of CMS information distribution strategy. 

Methods 

The population and sample, research design, outcome measures, and data analysis procedures are described in this 
section. 

Population and Sample 

The target population for this study was students using CMSs in higher education. Graduate students at a large 
mid-Atlantic university were the experimentally accessible population. The participating university had 
implemented a CMS in 2001. 

The sample was comprised of 52 students enrolled in 3 sections of a graduate-level research design course. 
This 14-week course met one evening per week and is a required course in three graduate programs: teaching and 
curriculum, health education, and training and development. Of the 52 students, 13 were enrolled in the Fall 2002 
section, 20 were enrolled in the Spring 2003 section, and 19 were enrolled in the Fall 2003 section. 

As shown in Table 1, data were collected on 10 attribute variables: (a) age, (b) gender, (c) educational level, (d) 
past academic coursework in research design, (e) graduate program in which currently enrolled, (f) Internet usage 
rate, (g) relevant prior knowledge of concepts and principles in research design, (h) learning style, (i) cognitive style, 
and (j) self-directedness. The last three variables (learning style, cognitive style, and self-directedness) were 
included in the study because they have been found to influence a person’s decision about the sources and methods 
that they use to learn something new. 

Several instruments were used to collect these data. Six items were included at the end of a research design 
knowledge test to collect data on students’ age, gender, educational level, past academic coursework in research 
design, graduate program in which currently enrolled, and Internet usage rate. Kolb’s Learning Style Inventory was 
used to assess students’ learning styles. This inventory is a statistically reliable and valid 12-item assessment tool 
that identifies four learning styles: converger (prefers to learn by solving problems and doing concrete tasks), 
accommodator (prefers hands-on, people-oriented learning activities), diverger (prefers to learn by observing, 
brainstorming, and gathering information) and assimilator (prefers to learn by putting 
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Table 1. Attributes of Students in Courses where Materials were Distributed in Class versus a CMS 









Materials Distributed via CMS 




Attribute Variables 


Materials Distributed in Class 


At Beginning 
of Semester 


1 Week Prior to 
Each Class 




Nominal data 


n 


% 


n 


% 


n 


% 


X 2 (2,N=52) 


Cognitive style 














1.194 


Field dependent 


4 


30.8 


10 


50.0 


8 


42.1 




Field independent 


9 


69.2 


10 


50.0 


11 


57.9 




Educational level 














0.695 


Bachelors 


13 


100 


19 


95.0 


18 


94.7 




Masters 


0 


0 


1 


5.0 


1 


5.3 




Gender 














1.820 


Male 


3 


23.1 


5 


25.0 


8 


42.1 




Female 


10 


76.9 


15 


75.0 


11 


57.9 




Graduate program 














9.867 


Health Education 


2 


15.4 


5 


25.0 


0 


0 




Teaching & Curriculum 


4 


30.8 


3 


15.0 


4 


21.1 




Training & Development 


6 


46.2 


12 


60.0 


15 


78.9 




None listed 


1 


7.7 


0 


0.0 


0 


0 




Learning style 














4.809 


Accommodator 


4 


30.8 


9 


45.0 


6 


31.6 




Diverger 


1 


7.7 


3 


15.0 


5 


26.3 




Assimilator 


6 


46.2 


4 


20.0 


4 


21.1 




Converger 


2 


15.4 


4 


20.0 


4 


21.1 




Past research design courses 














4.337 


Yes 


5 


38.5 


2 


10.0 


3 


15.8 




No 


8 


61.5 


18 


90.0 


16 


84.2 




Interval data 


M 


SD 


M 


SD 


M 


SD 


F 


Age 


35.6 


8.60 


35.4 


8.57 


35.5 


8.74 


0.003 


Internet usage rate a 


3.6 


1.06 


4.2 


0.95 


4.3 


0.87 


1.405 


Prior research design knowledge b 


8.3 


1.80 


7.8 


1.88 


8.1 


2.22 


0.259 


Self-directedness c 


238.9 


26.94 


226.3 


19.28 


227.5 


27.02 


1.214 



Notes 

The chi-square and ANOVA tests found no significant differences among the 3 treatment groups on any of the 10 attribute variables at p < .05. 
a Scale for Internet usage rate: 1 (Never) to 5 (More than 5 times per week) 
b Total possible points for the research design knowledge pretest =15. 
c Total possible points for the self-directedness assessment = 260. 



information into concise logical order) (Kolb, 1999). These four learning styles provide insights about one’s 
preferences for solving problems, working in teams, and managing personal and professional relationships. 

The Group Embedded Figures Test (GEFT) was used to assess a cognitive style called field 
dependence/independence ( Group Embedded Figures Test, 1971). The validity and reliability of the GEFT have 
been extensively documented. The assessment activity involves the identification of 18 simple geometric shapes 
that are embedded in complex figures; more field independent individuals tend to be more successful at identifying 
the shapes embedded within the complex figures. Previous studies have found that field-independent learners are 
more autonomous, better problem solvers, and more likely to succeed academically than field-dependent learners 
(Pithers, 2002). 

The Self-Directed Learning Readiness Scale was used to assess self-directedness. This is a 58-item self-report 
instrument, designed to measure the complex of attitudes, abilities, and characteristics which comprise readiness to 
engage in self-directed learning (Guglielmino, 1995). This instrument has been widely used to examine the self- 
directedness of adults in professional areas such as medicine and business. Its validity and reliability have been 
extensively documented (Guglielmino, 1996). Total possible scores on the SDLRS range from 0 to 290. 

Research Design 

A non-equivalent control group design was used to examine the effects of three information distribution 
strategies in a CMS on student performance and satisfaction (McMillan, 2004). With this design, pretests on 
research design knowledge, cognitive style, learning style, and self-directedness were administered in the first class 
of the semester for each of the three treatment groups. In the ensuing 13 weeks of the semester, class notes and 
handouts were distributed to students using three different strategies. The first information distribution strategy 
involved providing students with all instructional materials in class and was implemented in the Fall 2002 semester. 
Only two features of the CMS were used with this class, the syllabus was posted on-line and the e-mail function was 
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used to facilitate communication between the instructor and students. Because in-class distribution of course 
materials has been considered the traditional way of sharing course information, this treatment group was considered 
the control group. The second information distribution strategy involved posting class notes and handouts for all 13 
class sessions on the CMS at the beginning of the semester. This strategy was implemented in the Spring 2003 
semester. The third information distribution strategy, implemented in Fall 2003, was similar to the second strategy 
in that class notes and handouts were posted on the CMS. However, in this treatment group the materials were 
posted on the CMS only one week prior to the class session in which they were presented. At the end of the 
semester, posttests on research design knowledge and reactions toward the instructional experience were 
administered and the total number of points that each student earned in the course was computed. 

Participation in the study was voluntary. None of the research study assessments factored into the grade 
students earned in the course. All 52 students enrolled in the three sections of the course agreed to participate in the 
study. 

The instructional objectives, content, and activities were kept constant for all three treatment groups. The same 
instructor taught all three courses and kept a journal to record any questions, events, or issues that occurred in the 
three classes that may have related to the research study. 

Outcome Measures 

Two dependent variables were examined in this study: student performance and student satisfaction with the 
instructional experience. 

Student performance. Two assessments were used to evaluate student performance. First, a 15-item multiple- 
choice test was used to assess student knowledge of research design concepts and principles. The test items were 
selected from the instructor’s manual of the research textbook used in the course (McMillan, 2004). Total possible 
scores on the test ranged from 0 to 15. A coefficient of stability was established through a test-retest procedure. In 
this procedure, 28 graduate students in 2 graduate classes during Summer 2002 took the test two times, two weeks 
apart. This procedure yielded a Pearson product-moment correlation coefficient of .73. The internal consistency of 
the test was also calculated, with a split-half correlation analysis producing a reliability coefficient of .71. 

The second performance assessment was the total number of points earned in the course. Grades were based on 
the total number of points students earned during the semester on two research article critiques (110 points) and two 
multiple-choice exams (125 points), with the total possible points equaling 235. 

Student satisfaction. Satisfaction with the instructional experience was assessed with a written survey. The 
survey asked students to indicate the degree to which they agreed with 16 statements concerning the content, design, 
instructor, and perceived impact of the course. In addition, one of the survey items asked students to indicate the 
degree to which they perceived that the use of technology supported their learning in the course. The survey used 
5-point Likert scales ranging from 1 (strongly disagree) to 5 (strongly agree). The validity and reliability of this 
survey instrument have been established in previous studies that also assessed student satisfaction with an 
instructional experience (Lohman & Finkelstein, 2000, 2002). 

Data Analysis Procedures 

Data were collected at the beginning of the semester on 10 attribute variables (age, gender, educational level, 
past research design coursework, knowledge of relevant research design concepts and principles, graduate program 
in which enrolled, Internet usage rate, learning style, cognitive style, and level of self-directedness). These data 
were analyzed to determine whether the three treatment groups differed significantly on any of the 10 attribute 
variables. If significant differences were found among the groups on any of these attribute variables, that variable 
would be included in the statistical analysis as a covariate and a multiple analysis of covariance would be used to 
test for differences among the three treatment groups on student performance and satisfaction. If no significant 
differences were found on any of the attribute variables, then a multiple analysis of variance (MANOVA), using 
posttest scores on student performance and satisfaction, would be used to test for significant differences among the 
three groups. 

Results 

The statistical tests of the attribute variables and of the two main research hypotheses are presented in this section. 
An alpha level of .05 was established a priori for these statistical tests. 

Analysis of the Attribute Variables 

Statistical tests were conducted to examine whether there were significant differences among the three treatment 
groups on any of the 10 attribute variables. As shown in Table 1, no significant differences were found among the 
three treatment groups on any of these variables. Therefore, it was concluded that none of the attribute variables had 
influenced the results of the study and none was included as a covariate in the tests of the research hypotheses. 
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MANOVA was used to examine whether any significant differences existed among the three groups on student 
performance and satisfaction. 

Tests of the Research Hypotheses 

Student performance. The average score on the research knowledge posttest for students receiving course 
materials in class was 12.7 (SD = 0.63), as compared to an average of 1 1.1 (SD = 2.20) for students receiving course 
materials via the CMS at the beginning of the semester, and 11.3 (SD = 2.05) for those receiving course materials 
via the CMS one week prior to each class session. A MANOVA revealed a statistically significant difference 
among the treatment groups on the research design posttest scores, F (2, 49) = 3.218, p = .048, with the students who 
received course materials in class scoring significantly higher than either group of students who received course 
materials via the CMS. 

In addition, the average number of total points earned by students in the course was 211.5 (SD = 16.42) for 
those receiving course materials in class, 213.9 points (SD = 18.03) for those receiving course materials via the 
CMS at the beginning of the semester, and 208.6 points (SD = 18.91) for those receiving course materials via the 
CMS one week prior to each class session. A MANOVA revealed no significant difference among the three 
treatment groups on this variable, F (2, 49) = 0.892, p > .05. 

Student reactions. An analysis of the reaction survey data showed that students in all three treatment groups 
had positive reactions toward the instructional experience. On a scale of 1 to 5, mean scores on the 16 survey items 
ranged from a low of 3.8 for students who received instructional materials via the CMS at the beginning of the 
semester on their expectations for applying what they learned in the course to future professional responsibilities, to 
a high of 4.9 for students who received instructional materials in class on their satisfaction with the instructor. A 
MANOVA found no significant differences among the three treatment groups on any of the 16 survey items. 

Discussion 

Analysis of the data found that the distribution of course materials in class resulted in significantly higher student 
performance on an end-of-the-semester objective test of research design knowledge than was found when the 
materials were distributed via a course management system. Flowever, no significant differences were found among 
the three information distribution strategies on total points earned in the course or on student reactions toward the 
instructional experience. Possible explanations for these findings and implications of the findings for theory and 
practice as related to utilizing a CMS in instructional programs are provided in this section. 

A possible explanation for the higher scores on the knowledge test for those that received course materials in 
class as compared to via a CMS relates to the principle of instruction dictating that new information should be 
presented in logical chunks during instruction to enhance learners’ motivation to engage in planned learning 
activities and to help them focus on the learning tasks at hand (Joyce & Weil, 1996; Smith & Ragan, 1999). In the 
current study, when course materials were available via the CMS, students had the opportunity to preview new 
course information at any time they desired. In contrast, when course materials were distributed in class the 
instructor adjusted the rate of information distribution in accordance with the pace of learning in the class. 
Additionally, as materials were distributed in class the instructor was able to explain their relevance to the lessons at 
hand. This continuous adjustment of the pace and timing of the in-class distribution of course materials may have 
enhanced students’ levels of motivation and ability to focus on key aspects of the lessons and, ultimately, resulted in 
greater acquisition of research design knowledge. 

Yet, no significant differences were found among the three treatment groups on the second measure of student 
performance, total course points. A plausible explanation for this finding was that the evaluative activities for which 
course points were earned required higher levels of thinking than the research design knowledge test did (Kemp et 
al., 1998). Students earned course points for their performance on two multiple-choice exams and two research 
article critiques. The second exam and both research article critiques required students to make critical judgments 
about the research designs and procedures that had been used in published research studies. These higher-level 
cognitive activities may have provided students, regardless of which treatment group they were in, with a better 
opportunity to demonstrate their learning than was the case with the knowledge test. 

A puzzling finding in the current study was that there were no significant differences among the treatment 
groups on student satisfaction with the instructional experience. It has been widely reported that the vast majority of 
students believe that accessing course materials on-line is convenient and helps them organize and prepare for class. 
However, students who accessed course materials via the CMS in the current study did not perceive that the use of 
technology or availability of course materials was substantially better than students who received materials in class. 
Interestingly, several students in the in-class treatment group indicated at the beginning of the semester that they 
wanted course materials (e.g. class notes and PowerPoint presentations) to be posted on-line. When the instructor 
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explained that the CMS would not be used for that purpose, students seemed mildly disappointed. However, that 
disappointment dissipated within a few weeks and at the end of the semester the ratings of the in-class treatment 
group on the quality of the instructional experience, including the utilization of technology in instruction, were 
similar to both CMS groups. 

When considering the implications of this study’s findings, several of its limitations should be noted. The 
treatment was administered in three sections of the same course over a three-semester period. This design enabled 
the use of one instructor for all three treatment groups, thereby minimizing the likelihood that different teaching 
styles or teaching ability levels influenced the results of the study. However, two inherent limitations of this 
research design required actions to be taken by the researcher to enhance the confidence that the results were due to 
the treatment and not to something else. The first limitation was that three intact student groups participated in the 
study. To examine whether this limitation impacted the study’s results, a statistical analysis of 10 attribute variables 
was conducted. No significant differences among the three groups were found on any of the 10 variables. A second 
limitation was that the three student groups experienced the treatment at different points in time. As a consequence, 
it was possible that events may have occurred in one of the semesters that influenced the performance and/or 
satisfaction of students in that group. A journal was kept by the instructor to record any such events in class 
sessions, on campus, or in the news that related to the integration of technology in the instructional process. Only 
one event was observed during the administration of the treatment that pertained to the use of CMS. Because 
inclement weather caused the last class session of the in-class treatment group to be cancelled, a research article that 
was to be used for the final exam was distributed to students via the CMS rather than in class. Despite these 
limitations, the findings of this study have important implications for theory, practice, and the future research of 
CMSs in relation to promoting student performance and satisfaction. 

Theoretical Implications for Integrating CMSs in Instruction 

The current study’s findings provide empirical evidence to extend understandings of Bonk’s ten-level Web- 
integration continuum (Bonk et ah, 1999). Specifically, this study’s findings suggest that higher levels of student 
performance and satisfaction will not occur if instructors continue to utilize CMS tools that pertain only to the first 
five levels of the continuum. The five levels in the lower half of the continuum focus on tools that provide students 
with alternative sources for course-related information. This is a critical aspect of the Web-integration continuum 
because most instructors have NOT extended their use of CMSs beyond these five levels (Young, 2004). This was 
the level at which the CMS in the current study was used - as a means of providing an alternative way for students 
to access course materials - and it did not lead to substantive improvements in student performance or satisfaction. 

As a consequence, it is reasonable to suggest that the second half of the Web-integration continuum (Bonk et 
al., 1999) is where the most promise and opportunity resides for utilizing a CMS to enhance student performance 
and satisfaction. Levels 6-10 of the continuum are distinguished from the first five levels in that the learning 
activities and tasks in the upper half of the continuum contain requirements of a course that are not provided 
elsewhere. These levels range from Level 6, where substantive and graded Web activities are provided, to Level 10, 
where the course fits within larger programmatic Web initiatives. Clearly, this study’s findings have shown that 
using a CMS to replicate learning tasks and activities performed in class does not lead to superior learning or 
affective outcomes. Yet, the question still remains as to the potential benefits to be derived from using a CMS to 
accomplish learning tasks and activities that go beyond those conducted in face-to-face classroom settings. 
Implications for Using CMSs in Instruction 

Implications of this study’s findings for practice can be interpreted in two ways. The first way is to assert that 
the study’s findings provide no evidence to warrant using CMS in the instructional process. CMSs cost millions of 
dollars to purchase and maintain. Instructors must spend countless hours learning how to use the technology, 
preparing materials for CMSs, and monitoring student use of course websites. And, for what benefit? This study 
found no evidence that CMS yielded any positive dividend in terms of student learning or satisfaction. 

On the other hand, a second way of interpreting the study’s findings is to assert that current instructional 
practice is underutilizing the technological and pedagogical capabilities of CMSs. This assertion implies that 
instructional practice must move beyond using CMSs to replicate learning activities and materials that are already 
provided in class. Instead, CMSs should be used to conduct learning activities that supplement in-class activities as 
well as to provide additional learning resources that are not distributed in class. In other words, pedagogical practice 
must move into the upper half of the Web-based continuum for integrating technology into instruction (Bonk et al., 
1999). 

However, instructors cannot be expected to move higher in the technology-integration continuum without 
additional time and technical assistance. Instructors’ teaching loads would need to be temporarily reduced to 
provide the time that is necessary for developing Web-based learning materials and activities as well as for learning 
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how to use CMS tools. In addition, support staff with expertise in designing instruction for Web-based environments 
need to be available to assist instructors with these critical instructional design tasks. 

Implications for Future Research of CMSs 

Further research in two areas would help clarify the potential value of CMSs in higher education. First, because 
the current study was conducted with graduate students, replications of the study with undergraduate students would 
provide a greater understanding of its generalizability to student populations in higher education settings. Second, 
future research studies must investigate the effects of integrating CMSs at the higher levels of Bonk’s Web- 
integration continuum on student performance and satisfaction. This is a crucial area of research because, at the 
present time, there is no empirical justification to advocate continuing the conventional use of course management 
systems as an alternative means of distributing course information and materials. 

Contributions to New Knowledge in HRD 

The study provided empirical evidence that refuted anecdotal claims that the use of CMS content-presentation tools 
leads to higher levels of student performance and satisfaction. This is vital information for many HRD instructors 
and practitioners who dedicate a substantial amount of time and resources to the integration of CMSs into their 
college courses and training programs. To promote higher levels of student performance and satisfaction, the results 
of the current study clearly indicated that instructional practice must utilize more sophisticated CMS tools to help 
instructors facilitate learning activities and tasks that are not provided elsewhere. 
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